Hypervideo information retrieval usingmultimedia

ABSTRACT

Disclosed is a method and device for selecting documents, such as Web pages or sites, for presentation to a user, in response to a user expression of interest, during the course of presentation to the user of a document, such as a video or audio selection, whose content varies with time. The method takes advantage of information retrieval techniques to select documents related to the portion of the temporal document in which the user has expressed interest. The method generates the search query to use to select documents by reference to text associated with the portion of the temporal document in which the user has expressed interest, as by using the closed caption test associated with the video, or by using speech recognition techniques. The method further uses a weighting function to weigh the terms used in the search query, depending on their temporal relationship to the user expression of interest. To conserve resources, the method does not transmit the closed caption or other synchronized multimedia information to the user, and obtains the necessary information about the temporal occurrence of terms in the temporal document from a database.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is related to the following co-pending applications:

Application Title Ser. No. Filing Date Hypervideo: Information RetrievalUsing 09/430,437 October 29, Time-Related Multimedia 1999 Hypervideo:Information Retrieval Using 09/430,597 October 29, Text from Multimedia1999 Hypervideo: Information Retrieval at User 09/429,731 October 29,Request 1999 Hypervideo: Information Retrieval Using 09/431,293 October29, Real Time Buffers 1999

All of which are commonly owned by the assignee of the instantapplication.

TECHNICAL FIELD

This invention relates to techniques for retrieving material on theWorld Wide Web, and more particularly to methods of retrieving Web pagesof interest to a user which relate to temporal material such as videoprogramming.

BACKGROUND OF THE INVENTION

The Internet, of which the World Wide Web is a part, includes a seriesof interlinked computer networks and servers around the world. Users ofone server or network connected to the Internet may send information to,or access information on, other networks or servers connected to theInternet by the use of various computer programs which allow suchaccess, such as Web browsers. The information is sent to, or receivedfrom, a network or server in the form of packets of data.

The World Wide Web portion of the Internet comprises a subset ofinterconnected Internet sites which may be characterized as includinginformation in a format suitable for graphical display on a computerscreen. Each site may include one or more separate pages. Pages, inturn, may include links to other pages within the site, or to pages inother Web sites, facilitating the user's rapid movement from one page orsite to another.

A number of the sites and pages accessed through the Web may consistentirely of “static” displays of text and/or images. These displays mayreside on one or more host servers or networks, and may be accessedthrough the Internet for storage and/or display on a remote server ornetwork. Other sites or pages may have changing advertisements or othersimilar material as well as “static” displays of text and/or images.

There are a number of techniques for permitting a user, while viewingone page or site on the Web, to request and be given access to othermaterial that relates to the material being viewed, which can be appliedwhen the material being viewed contains static text or image displays inwhole or in part.

In addition to accessing static displays of text and/or images on theWeb, it also may be possible to access material on the Web which isdynamic or changing. Such material will be referred to as “temporaldocuments” to reflect the fact that, unlike static material, theircontent as made available to or perceived by a user may change with thepassage of time. Examples of such temporal documents are multimediamaterial such as video and audio programming, but there are other typesof temporal documents as well. For example, the text of news bulletins,stock quotations such as would be seen on a “ticker tape”, or sportsscores may be made available; material such as this by its nature alsomay be changing as it is viewed, either because the underlyinginformation is changing, or because the information is “scrolled” acrossthe user's monitor, thus appearing as constantly changing with time.Other types of dynamic or changing material will also be apparent to oneof ordinary skill in the art.

Temporal documents may have been previously created and stored on aserver for later access (such as a movie, or a recording of apreviously-occurring sports event) or a temporal document may reflect anevent that is occurring “live” at the time the temporal document istransmitted over the Internet (such as a live news broadcast or sportsevent, or a stock ticker displaying real-time stock transactioninformation).

Whether the temporal document is previously-created or is being accessedlive, it is useful to have a technique to facilitate a user obtainingmaterial that relates to a portion of the temporal document he isviewing or listening to. Because the material is changing, however, someof the techniques that may be used to provide access to material that isrelated to a static page being viewed, may not be readily applicable totemporal documents.

Some previous methods of providing additional material related tochanging content such as video programming have relied upon the priormanual choice of other Web documents, such as pages or sites, to beassociated with particular portions of the video content. Then, when aparticular portion of the video programming is reached, the related Webpage or document may automatically be presented to the user, or the usermay be informed of the availability of a link to the related material,and offered the choice of accessing it. Alternatively, no informationabout related material may be presented until or unless a user indicatesinterest during a particular segment of the video material (as by“clicking” with a mouse, or pressing a button); upon an expression ofinterest, the particular other Web page previously chosen as related tothis portion of the video material may be presented to the user.

This method of associating related material to a temporal document hasdrawbacks, however. Because it requires the preselection of theassociated material, it cannot be utilized with live material, or withmaterial that has not been previously analyzed for the purpose. It alsomay be costly, in that it may require intensive manual manipulation ofthe multimedia material to choose other Web pages to associate with eachportion of the video or audio material, and to carry out theassociation. It also may be rigid, in that once the selection is made itmay remain unchanged regardless of whether other more appropriaterelated material becomes available. It may be both expensive andtime-consuming to make changes once links have been established.Additionally, this method may offer a very limited choice to the user inthat it may not be practical to offer a large number of links at eachportion of the video or audio material.

Thus, there is a need for a method or device for permitting a user toobtain access to other material that is related to a portion of atemporal document (such as a video or audio program) being accessed onthe Web, where the selection of the related material offered to the useris not made in advance, but is done automatically at the time the userexpresses an interest in obtaining such material. Such a method ordevice makes “hypervideo” a practical concept.

One aspect of this need is a need for determining the portion of thetemporal document about which the user would like to obtain additionalinformation. In the case of a “static” display of material as might bepresented to the user on a computer monitor, it may be possible to havethe user indicate the material of interest by using a mouse or othersimilar selection device to maneuver a cursor on the monitor until it issuperimposed on the portion of the display of interest, and then to“click” on the material of interest. In the case of a changing display,such as video, that may not be practical. For example, because it maytake a certain amount of time for the user to decide that he isinterested in obtaining additional material, and a certain amount oftime to maneuver the mouse or other signaling device to indicateinterest, the expression of interest may be delayed by a certain amountfrom the actual material in which the user is interested.

Another aspect of this need is a need for determining what othermaterial is related to the material in which the user has expressed aninterest. In the case of a static display which includes a display oftext, it may be possible to have the user indicate the specific materialin which he is interested (as by using a mouse to maneuver a cursor tothe word or term displayed on the screen), and then to use that specifictext as the basis of a search query using a conventional Web searchengine. But in the case of video material, that may not be possible.

BRIEF DESCRIPTION OF DRAWINGS

The above-mentioned and other features of the invention will now becomeapparent by reference to the following description taken in connectionwith the accompanying drawings in which:

FIG. 1 is a schematic diagram of an embodiment of a computer system thatmay be operated according to the present invention.

FIG. 2 is a diagram illustrating the weight to be assigned to differenttemporal portions of material such as video, based upon a user responseat time t₀, according to one embodiment of the present invention.

FIG. 3 is a diagram illustrating the weight to be assigned to differenttemporal portions of material such as video, based upon a user responseat time t₀, according to another embodiment of the present invention.

FIG. 4 is a diagram illustrating the weight to be assigned to differenttemporal portions of material such as video, based upon a user responseat time t₀, according to a further embodiment of the present invention.

FIG. 5 is a diagram illustrating the weight to be assigned to differenttemporal portions of material such as video, based upon a user responseat time t₀, according to a further embodiment of the present invention.

FIG. 6 illustrates a conventional (prior art) relationship betweendocuments and inverted term lists.

FIG. 7 illustrates conventional (prior art) lookup tables which may beused in conjunction with inverted term lists.

FIG. 8 illustrates a relationship between documents and compresseddocument surrogates

FIG. 9 is a flow chart which illustrates a process by which a documentscore may be calculated, using compressed document surrogates.

FIG. 10 is a flow chart which illustrates a process by which a searchquery may be carried out to identify material relating to a portion of atemporal document in which a user has expressed an interest, usingcompressed document surrogates according to the present invention.

SUMMARY OF THE INVENTION

According to the present invention, finding documents which relate to aportion of a temporal document includes (a) in response to a signal ofinterest at a particular time during the temporal document, identifyinga portion of the temporal document for which related documents are to befound, (b) selecting text associated with the portion of the temporaldocument identified, by reference to a database wherein is stored eachterm in text associated with the temporal document, together with a timein the temporal document at which the term occurs, (c) weighting eachterm in the text selected by a function according to the time at whichthe term occurs relative to the time at which the signal of interestoccurs, and (d) finding the related documents by use of informationretrieval techniques applied to the text selected. The temporal documentmay be video or audio material. The video material may be stored on avideo server. The closed-caption text associated with the temporaldocument may not be made available when the temporal document isaccessed from the video server. The text stored in the database may bedetermined by the application of speech recognition techniques to theaudio component of the portion of the temporal document identified, ormay be the closed-caption text associated with the portion of thetemporal document identified. The temporal document may include text,which text appearing to the user may vary with time, and the textselected may be that portion of the temporal document identified. Thetext may include news bulletins, weather, sports scores or stocktransaction or pricing information. The weighting function W(t) may beequal for all times between a time t₁ before the signal of interest isgiven and a time t₂ before the signal is given, and may be zero for allother times. T₁ may be 2 seconds and t₂ 30 seconds. Alternatively, W(t)may be equal for all times between a time t₁ before the signal ofinterest is given and a time t₂ before the signal is given, and maydecrease from t₁ until the time of the signal, and increase from a timet₃ before the signal is given to the time t₂, and may be zero for allother times. T₁, t₂ and t₃ may have specified values. W(t) may decreaselinearly from the time t₁ until the time of the signal, and increaselinearly from the time t₃ before the signal is given to the time t₂.W(t) may be a double exponential function of time. W(t) may equal 0 forall times earlier than 30 seconds before the signal of interest isgiven, and later than the signal of interest. The related documents maybe accessed through the Internet. The related documents may be selectedfrom among a collection of documents which may be accessed through theInternet, by utilizing databases comprising information about thecollection. The related documents may be selected from the collectionaccording to the scores achieved when evaluating documents in thecollection according to a formula giving scores to documents dependingupon the occurrence in the documents of terms which occur in textassociated with the portion of the temporal document identified, whereeach term is weighted by the weighting function according to the time atwhich the term occurs relative to the time at which the signal ofinterest occurs. The terms which occur in the text which is associatedwith the portion of the temporal document identified, and the times atwhich the terms occur, may be obtained by reference to the databasewhich contains the terms and the times in the video or audio material atwhich the terms occur. The times may be determined and stored in thedatabase using the synchronization information contained in the temporaldocument. The temporal document may use the SMIL standard forsynchronization of multimedia elements in a document. A predeterminednumber of documents, 1000, may be selected. A score S_(D) of a documentD in the collection may be determined by crediting the document D, foreach term T in the temporal portion of the document identified whichoccurs in the document D, with an amount proportional to the weightingfunction, Robertson's term frequency TF_(TD) and to IDF_(T). Terms whichoccur in portions of the temporal document other than the portionidentified may be utilized in calculating scores achieved whenevaluating documents in the collection. The determination of thedocuments in the collection which receive the highest scores may becarried out using compressed document surrogates. The determination ofthe documents in the collection which receive the highest scores may becarried out by a server which is distinct from the server which receivesthe signal of interest.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Referring to FIG. 1, a computer system 1 includes a workstation 2 havinglocal storage 3. The workstation may also be connected to a local areanetwork 4 and may access to the Internet 5. The Internet 5 may includeor be coupled to remote storage 6. The workstation 2 may be any one of avariety of commercially available computers capable of providing thefunctionality described in more detail below. The local storage 3 mayinclude ROM, RAM, a hard disk, a CD, or any other media capable ofcontaining data and/or programs for the workstation 2 or other data. Thelocal area network 4, which is coupled to and exchanges data with theworkstation, may also contain data and/or program information for use bythe workstation 2. The Internet 5 may be accessed in a conventionalmanner by the workstation 2. Alternatively, the workstation 2 may accessthe Internet 5 through the local area network 4, as shown by the dottedline of FIG. 1. The remote storage 6 may also contain data and/orprogram information for the workstation 2 or may contain otherinformation, as will become apparent from the description below.

The system described herein permits a user (utilizing the computersystem 1 which includes the workstation 2) who has accessed the Internet5, either directly or through the local area network 4, to be givenaccess to other material that is related to a temporal document, such asbut not limited to video or audio material, the user is accessing. Inone embodiment, the system includes software written in the Javalanguage, running on a Hewlett Packard server connected to the Internet,as well as software written in the C language and in PERL running on anSGI O2 server connected to the Internet. Of course, it will beappreciated by one of ordinary skill in the art that the system may beimplemented using a variety of computers and programming languages.

The system may be accessed by the user through the Internet 5 from hisworkstation 2 using a Web browser of conventional design, as would befamiliar to one of ordinary skill in the art. The user then accesses atemporal document. In one embodiment, the temporal document is obtainedfrom a collection of temporal documents previously prepared by thesystem and placed in a video library made available through a videoserver maintained in connection with the system. In this embodiment, theuser may be permitted to choose the document in any one of a number ofways which will be known to one of ordinary skill in the art. The usermay be given a list of documents which are available, and permitted tochoose one, by clicking on it or indicating his interest in any one of anumber of alternative ways which will be known to one of ordinary skillin the art. Alternatively, the user may be invited to search by usingsearch engine or search query techniques such as will be familiar to oneof ordinary skill in the art. Still other methods to permit the user tochoose a document from among those in the library will be known to oneof ordinary skill in the art. The user then may view (or listen to) thetemporal document chosen through his work station 2 connected to theInternet 5.

In another embodiment, the temporal document may be obtained fromanother source on the Web. In this embodiment, the user may be permittedto employ a search engine which is maintained as part of the system tofind and retrieve a document to the system. The search engine employedmay be any one of a number of a type which will be familiar to one ofordinary skill in the art. The user then may view (or listen to) thetemporal document chosen through his work station 2 connected to theInternet 5.

In another embodiment, the temporal document may be obtained fromanother source on the Web. In this embodiment, the user may be permittedto employ a search engine on his work station 2 connected to theInternet 5 to retrieve and view (or listen to) the temporal documentchosen. The search engine employed may be any one of a number of a typewhich will be familiar to one of ordinary skill in the art. The userthen may view (or listen to) the temporal document chosen through hiswork station 2 connected to the Internet 5.

The system utilizes IR (information retrieval) techniques to select therelated material when interest in having access to such material isindicated. The system analyzes the content of a portion of the temporaldocument as to which the interest has been indicated, rather thanpre-storing links to material which is determined to be related inadvance.

The system may be utilized in connection with any material which has acharacteristic that, when accessed by a user or viewer through thecomputer system 1 which may include the workstation 2, it changes withtime. This includes but is not limited to video material and audiomaterial, such as movies, news programs, and sports events. It may alsoinclude, for example, textual news bulletins that are displayed, eitheralone or superimposed on other content, or stock quotations or sportsscores. These materials may be changing with time in that they arescrolled across the monitor for reading purposes, so that the portion ofthem accessed by the user changes with time.

If the material accessed is video material, whether collected into avideo library and previously stored in a video server, or accessed fromanother location on the Internet, the video material may have beenpreviously broadcast, and each video may have associated therewithclosed captions which contain text that accompanies the video. Theclosed caption material may include the text of dialogue, or spokenwords that accompany the video and constitute the audio track.

Included in the system is a technique that may be used to indicate whena portion of the temporal document in which there is interest has beenreached. That a portion of the temporal document as to which additional,related material is desired has been reached, is indicated by means of aparticular, preselected response being made after the portion of thedocument is displayed to the user. In one embodiment, a mouse isclicked, while in other embodiments software which recognizes andresponds to voice commands may be employed, a particular key (or anykey) on a keyboard may be depressed, or a button on a joystick may bepressed. Other methods of providing a signal to a computer system, knownto one of ordinary skill in the art, may also be utilized.

Further included in the system is a technique which may be used, when asignal indicating interest in a portion of the document is given, tofacilitate the determination of the portion of the temporal document inwhich the interest has been indicated, by utilizing the time at whichthe signal indicating interest has been given.

It is understood that a user may not be able to instantaneously thinkabout the changing material that is being presented, make a decisionthat he is interested, and give the required signal. Moreover, it isunderstood that while the user sometimes may make a decision aboutinterest based upon what appears or is heard at a particular instant, atother times the decision may be based upon a sequence of materialpresented over a period of time, rather than based upon the material ata particular instant.

For these reasons, the technique used in the system does not treat thecontent of the temporal document at the instant the signal is given asthat portion of the temporal document in which there is interest, andtherefore as a basis for finding related material. Rather, it is assumedthat there is a delay between the material of interest first beingpresented to the user, and the indication of interest, and it is furtherassumed that the user is interested in material which extends over aperiod of time. In particular, it is assumed that the interest of theuser in the content of the temporal document may be expressed as afunction W(t) of the time t prior to the signal indicating interestbeing given.

In one embodiment of the system, it is assumed that there arecharacteristic fixed delay times t₁ and t₂, such that the interest ofthe user in the content of the temporal document begins at time t₂before the indication of interest and ends at time t₁ before theindication of interest, and is equal between times t₁ and t₂,. A diagramof the interest as a function of time W(t) in this embodiment is shownin FIG. 2. While other values of t₁ and t₂ may be used without departingfrom the spirit and scope of the invention, in this embodiment t₁=2seconds and t₂=30 seconds.

In another embodiment, it is recognized that a more realistic model mayassume more gradual and probabilistic decisionmaking and responses. Inthis embodiment, rather than assuming that there is no interest in anycontent from after the time t₁, it is assumed that there is some butlesser interest in material between the time t₁ and the time at whichthe interest is expressed and that the interest decreases from the timet₁ to the time at which the interest is expressed In this embodiment, itis further assumed that there is some interest in content from earlierthan time t₂, starting at a time t₃, and that the interest increasesfrom the time t₃ to the time t₂. In this embodiment, it is furtherassumed that the interest may vary between times t₂ and t₁. A diagram ofthe interest as a function of time W(t) in this embodiment is shown inFIG. 3. While other values of t₁, t₂ and t₃ may be used withoutdeparting from the spirit and scope of the invention, in this embodimentt₁=2 seconds, t₂=15 seconds, and t₃=30 seconds.

In yet another embodiment, for simplicity it is assumed that theinterest in the content is equal between times t₁ and t₂, and it isassumed that the interest in content decreases linearly from the time t₁to the time at which the interest is expressed In this embodiment, it isfurther assumed that the interest in content increases linearly from thetime t₃ to the time t₂. A diagram of the interest as a function of timeW(t) in this embodiment is shown in FIG. 4. While other values of t₁, t₂and t₃ may be used without departing from the spirit and scope of theinvention, in this embodiment t₁=2 seconds, t₂=15 seconds, and t₃=30seconds.

In another embodiment of the system described herein, a discrete twostage exponential function is used to model the interest in content as afunction of time, for the time period prior to at which the interest isexpressed:P_(t1, t2)(t) = ∫_(k = 0)^(t)(1 − exp (−t₁)) * exp (−t₁k) * (1 − exp (−t₂)) * exp (−t₂(t − k))

While other values of t₁ and t₂ may be used without departing from thespirit and scope of the invention, in this embodiment t₁=0.0001 andt₂=0.00025, where time is expressed in milliseconds. A diagram of theinterest as a function of time W(t) in this embodiment is shown in FIG.5, where time is expressed in seconds.

In the system described herein, the determination of what material maybe related to the portion of the temporal document in which the user hasindicated an interest may be made by using text associated with thatportion of the temporal document which has been identified by means ofapplying the above aspect of the system.

The text to be utilized may be obtained in a number of ways. If thetemporal document itself comprises text, such as breaking news bulletinsdisplayed visually on a portion of the computer monitor, a portion ofthe text that is associated with the portion of the temporal documentwhich has been identified may be utilized. If the content includessymbols, such as stock prices displayed using abbreviations to identifythe company, a portion of the symbols that is associated with theportion of the temporal document which has been identified may beconverted to text, and the text utilized.

If the temporal document is a video or audio program, a number ofdifferent techniques may be utilized to obtain relevant text. In oneembodiment, text which results from the application of speechrecognition software to the portion of the audio program which has beenidentified, or the audio component of the portion of the video programwhich has been identified, may be used. Speech recognition software of akind familiar to one of ordinary skill in the art may be used.

In another embodiment, relevant text may be obtained by use of theclosed caption information which is associated with the portion of thevideo programming which has been identified. If this is done, and theoriginal video material was analog, the closed caption text may beextracted from the analog video by use of a commercially availableclosed caption decoder that will be familiar to one of ordinary skill inthe art such as that available from Link Electronics.

In the system described herein, if a collection of temporal documents ispreviously prepared by the system and placed in a video library to bemade available through a video server maintained in connection with thesystem, when the temporal documents are placed in the video library atable is created and stored for each temporal document which containseach term contained in the text of the document, in the order in whichthe terms occur in the text temporally, and associated and stored in thetable with each term is the time t at which the term occurs in thetemporal document.

If a temporal document utilizes the Synchronized Multimedia IntegrationLanguage (SMIL) standard for delivery of synchronized temporal media,the existing synchronization information in the document may be utilizedto extract the times at which the terms occur. The method of doing sowill be apparent to one of ordinary skill in the art. (SMIL is definedusing the XML standard and allows the layout of temporal media to bespecified, as well as the synchronization of multiple temporal mediastreams. SMIL provides synchronization elements whereby begin and endtimes as well as durations and synchronization points can be specifiedfor multiple media streams. The use of the SMIL synchronizationinformation allows the content of one stream, such as closed captiontext, that occurs contemporaneously with the content of another stream,such as video, to be extracted. The SMIL 1.0 specification may be foundat www.w3.org/TR/1998/REC-smil-19980615)

If the temporal document uses a synchronization method other than SMILfor its multimedia content, the synchronization information generated bythat method may be used to extract the times at which the terms in theclosed caption text occur.

If the original temporal document was video which was obtained in analogform, and it is desired to utilize the closed caption, acommercially-available closed caption decoder of a type familiar to oneof ordinary skill in the art, such as that available from LinkElectronics, may be used.

The text associated with the portion of the temporal document which hasbeen identified is used to locate other material that may be related tothat portion of the temporal document in which interest has beenindicated. This is done by using the associated text as a basis for asearch query on a database of documents located on the Web. Thedocuments in the database include but need not be limited to Web pagesor sites.

In order to improve the relevance of the material thus selected, a termin the text which occurs at a time t relative to the time at which theinterest has been indicated is weighted in the search query by thefunction W(t).

Depending upon the form of the function W(t), and other considerationswhich will be apparent to one of ordinary skill in the art, in order toreduce the time required to apply the search query it may be determinedto include only times t for which the function W(t) is greater than apredetermined quantity, or only times t within a specified time prior tothe indication of interest. In one embodiment, where the function W(t)is a discrete two stage exponential function in which time is expressedin milliseconds, and t₁=0.0001 and t₂=0.00025, only times t within 30seconds (30,000 milliseconds) before the indication of interest areincluded in the analysis.

In this embodiment, if the temporal document involved is one whichpreviously has been placed in a video library and made available througha video server maintained in connection with the system, the terms to beincluded in the search query are selected by consulting the table forthe temporal document which contains all terms in the text associatedwith the document, and the times at which the terms occur, and choosingall terms which occur within the 30 seconds before the indication ofinterest.

While other search query methods known to persons of ordinary skill inthe art may be utilized to find relevant material, in the preferredembodiment Robertson's term frequency score is employed.

In this embodiment, the search query is run on the collection ofdocuments from which the relevant material is to be drawn, and adocument D in the collection is given a score as follows:${S_{D} = {\sum\limits_{{terms}\quad T}{{W(t)}*{TF}_{TD}*{IDF}_{T}}}},$

where: S_(D) is the total score for a document D,

W(t) is the weight assigned to term T which occurs at time t

TF_(TD)=Robertson's term frequency for Term T in Document D

=N _(TD)/(N _(TD) +K ₁ +K ₂*(L _(D) /L ₀)),

where: N_(TD) is the number of times the term T occurs in document D,

L_(D) is the length of document D,

L₀ is the average length of a document in the collection of documentsindexed, and

K₁ and K₂ are constants. (K₁ typically may be assigned a value of 0.5,and K₂ 1.5, but these values may be varied without departing from thespirit and scope of the invention.) and IDF_(T)=log ((N+K₃)/N_(T))/log(N+K₄)

where:

N is the number of documents in the collection,

N_(T) is the number of documents containing the term T in thecollection,

K₃ and K₄ are constants. (K₃ typically may be assigned a value of 0.5,and K₄ 1.0, but these values may be varied without departing from thespirit and scope of the invention.)

This particular formula is by no means the only formula that may be usedto analyze documents for relevance. Other formulae will be apparent toone of ordinary skill in the art. For example, the weight to be assignedto a term in the search query may be adjusted depending on whether, andhow frequently, in relative or absolute terms, the term occurs in theportion of the temporal document which falls outside the time boundariesused for determining whether a term is to be included in the searchquery.

Documents are then ranked in order of their scores S_(D), and thehighest-ranking documents are returned to the user as relevant to theportion of the temporal document in which he has expressed an interest.(While any number of documents may be returned, in the one embodiment1000 is the maximum number that will be returned.)

The search may be carried out by the same server which has received thesignal from the user, selected the text which is to be utilized in thequery, and determined the weights to be assigned to each term in thetext by reason of its temporal relationship to the signal of interest.In one embodiment, however, the query is processed by an IR server,while the other functions—receipt of the signal of interest,determination of the text to be the query, and temporal weighting of thetext—are carried out by a separate QSE (query string extractor) server.

The documents in the collection which is utilized as the basis for theprocessing of the query may be selected for inclusion in the collectionby any one of a number of methods that will be familiar to one ofordinary skill in the art. For example, the documents may be selected bya processing of automatically spidering the Web and indexing pages andsites thus located and determined to meet predetermined criteria.Techniques for developing programs to spider the Web will be known toone of ordinary skill in the art. and are described for example in WebClient Programming in PERL, Clinton Wong, O'Reilly and Assoc., 1997. Forexample, only sites that relate to specific subjects, such as electroniccommerce, may be selected for inclusion in the collection, or only sitesjudged suitable for access by children of a certain age range. Thedocuments included in the collection could include (or could be limitedto) other video or audio materials, and/or text.

In processing the query, it is useful to take advantage of certain otheraspects of the system to make the search quicker and more efficient.These aspects respond to problems which arise out of the fact that manycommon schema for the retrieval of Web documents of interest (includingbut not limited to Web pages or sites) rely upon the use of invertedterm lists to maintain information about the use of various terms in thedocuments, but do not maintain information about the documentsthemselves, other than through the inverted term lists.

In order to understand these aspects, it is appropriate first todescribe the structure of a conventional inverted term list, and itsrelationship to the underlying collection of documents about which itcontains information. FIG. 6 illustrates one possible conventionalrelationship between underlying documents in a document collection, suchas, but not limited to, the Web or a portion thereof, and associatedinverted term lists which may be used to facilitate the retrieval ofdesired documents from the collection. Either Web sites or Web pages maybe treated as documents.

In constructing inverted term lists, it is useful to decide what termsshould be included. It may be determined to store information withrespect to all terms which occur in documents in a collection, or it maybe determined to exclude common words such as “the” and “and,” or it maybe decided to store information only about certain specified terms, suchas those which may occur in a particular field such as a scientific ortechnical discipline. (A term may be a word, a number, an acronym, anabbreviation, a sequential collection of the above, or any othercollection of numerals, letters and/or symbols in a fixed order whichmay be found in the documents in the collection to be searched.) Ingeneral, terms that are considered to be useful for purposes ofretrieving documents may be selected.

An inverted term list may be created for each term of interest that isfound to occur in any of the documents in the collection. In the exampleillustrated in FIG. 6, inverted term lists 835, 840, 845 identify, bymeans of providing a unique document identifier number, every documentfrom the collection in which corresponding terms 836, 841, 846 occur,and state how many times each of the terms 836, 841, 846 occurs in thedocument. Thus, in FIG. 6 the inverted term list 835 corresponding tothe term 836 states how often the term 836 occurs in each of thedocuments 805, 815, 825 in the collection. In this example, the invertedterm list 835 for the term 836 contains an entry for the unique documentidentifier number of the first document, “1”, and states that the term836 occurs twice in Document 1 805, then an entry for the uniquedocument identifier number, “2”, of the second document, and a statementthat the term 836 occurs once in Document 2 815, then an entry for theunique document identifier number, “3”, of the third document, and astatement that the term 836 occurs twice in Document 3 825, and so on.It will be appreciated by one of ordinary skill in the art that invertedterm lists may also contain other information as well, as will bediscussed below.

Inverted term lists may be stored as linked lists, or they may be fixedarrays. Other equivalents will be apparent to those of ordinary skill inthe art.

Lookup tables may be created in connection with inverted term lists. Onelookup table which may be created may provide the locations in thedocument collection of the documents whose contents have been indexed inthe inverted term lists; in the case of Web pages or sites, the URLs ofthe pages or sites may be provided. An example of such a lookup table100 is shown in the upper portion of FIG. 7. The document URLs may bestored in the lookup table in the order of the unique documentidentifier numbers of the documents. Then, if the inverted term listsinclude the document identifier numbers of the documents having the termin question, and the lookup table is maintained as a fixed array, thelocation in the lookup table array of an actual document URL may bedetermined directly from the document identifier number.

If such a lookup table is not created, inverted term lists may containthe locations in the document collection, such as the URLs, of thedocuments which contain the term in question.

Another lookup table may provide information about the terms for usewhen searches for relevant documents are done using the inverted termlists. An example of such a lookup table 102 is shown in the lowerportion of FIG. 7. For each term, this lookup table may contain theEnglish (or other natural language) term itself, the address of theinverted term list for the term, and other information which may be ofuse in using the inverted term lists to rank documents for relevance,such as, but not limited to, the number of documents in the collectionin which the term occurs, the number of times the term occurs indocuments in the collection, and the maximum term frequency score forthe term in any one document in the collection.

The term frequency scores for the term may be calculated based on anyone of a number of formulae which will be familiar to one of ordinaryskill in the art, such as but not limited to Robertson's term frequencyformula:

TF _(TD) =N _(TD)/(N _(TD) +K ₁ +K ₂*(L _(D) /L ₀)),

where N_(TD), L_(D), L₀, K₁ and K₂ have the values set forth above.

The terms may be stored in this lookup table in any order, such asalphabetical order. For ease of reference they may be stored in thenumerical order of unique term identification numbers assigned to eachterm. If this is done, and the lookup table is maintained as a fixedarray, the location of information about a term in the lookup table maybe determined directly from the term identification number of the term.

The inverted term lists also may contain the number of documents in thecollection in which the term occurs, the number of times the term occursin documents in the collection, and/or the maximum term frequency scorefor the term in any one document in the collection, if this informationis not maintained in the lookup table which contains the address of theinverted term list for the term. The inverted term list for a term alsomay contain, not simply the number of times the term occurs in aparticular document, but the location in the document at which the termoccurs.

A single inverted term list may be maintained for each term of interest.Alternatively, in order to permit more expeditious responses to searchqueries, two inverted term lists may be maintained for each term ofinterest. The first, or “top” inverted term list, may containinformation about an arbitrary number of documents, such as 1000, whichhave the highest term frequency scores for the term. The second, or“remainder” inverted term list, may contain information about theoccurrence of the term in the remaining documents. (If, separate top andremainder inverted term lists are maintained, then a lookup table 102which contains the maximum term frequency scores for terms may containseparate maximum term frequency scores for documents on the term's topinverted term list and for documents on the term's remainder invertedterm list.)

In the inverted term lists, information about documents may be stored inorder of the term frequency score for the documents, so that thedocuments with the highest term frequency scores are placed at the topof the inverted term list.

In order to facilitate execution of search queries using inverted termlists, a compressed document surrogate may be used for storinginformation about a document that is part of a collection of documentsof potential interest. This may be illustrated as applied to a casewhere the documents of interest are Web pages, but persons of ordinaryskill in the art will recognize that it may equally be applied tocollections of Web sites or of other varieties of computerizeddocuments.

As is the case in creating inverted term lists, it may be determined tostore information with respect to all terms which occur in documents ina collection, or it may be determined to exclude common words such as“the” and “and,” or it may be decided to store information only aboutcertain specified terms, such as those which may occur in a particularfield such as a scientific or technical discipline. If the compresseddocument surrogates are to be used in conjunction with inverted termlists, the same set of terms which the inverted term lists cover may beused in the compressed document surrogates. (Hereinafter, the set ofterms about which it has been determined to store information arereferred to as the “terms of interest.”)

If inverted term lists are not created for multiword terms, and theinverted term lists and compressed document surrogates do not maintaininformation about the location of terms in a document, but it is desiredto be able to search for multiword terms, the compressed documentsurrogates may include multi-word terms which are omitted from invertedterm lists. If this is done, a search for a multiword term may beperformed by searching for each word in the term, and then consultingthe compressed document surrogate of any document found to contain theindividual words, to determine if the desired multiword term is in thedocument.

A compressed document surrogate for a particular document comprises atable of desired information about all of the terms of interest whichoccur in the document, in a suitable order. This desired information mayinclude the number of times the term occurs in the document, and/or theterm frequency score for the occurrence of that term in that document,according to Robertson's term frequency formula or any other formula,and/or the location in the document (in absolute terms or relative tothe prior occurrence) of each occurrence. (Other relevant informationmay be added at the discretion of the user without departing from thespirit or scope of the invention.) Alternatively, a compressed documentsurrogate may simply indicate that a term occurs in the document, withno further information about specific occurrences or about the number ofoccurrences. A compressed document surrogate may provide the address ofthe inverted term list for each term of interest which occurs in thedocument, and/or the address of the location in the inverted term listof the entry for that document. Alternatively, a compressed documentsurrogate may provide the address of a location in a lookup table of aterm of interest which occurs in the document, or information, such as aterm identification number, from which the address of a location in alookup table of the term may be determined.

In the preferred embodiment of a compressed document surrogateillustrated in FIG. 8, it is determined to store information about allterms which occur in documents, other than specified common words. Inthis embodiment, it is further decided that a compressed documentsurrogate for a document shall identify each term of interest found inthe document, and specify how many times the term occurs in thedocument, but shall provide no further information about the occurrenceof terms in the document.

In this embodiment, the term information in the document surrogates isstored in order of term identification number. Each term is assigned aunique integer identification number. (Term identification numbers areassigned to terms in the order in which the terms are first encounteredin the course of constructing the table and associated inverted termlists, so that the first term found in the first document indexed isassigned the term identification number “1”, and so on. Since terms areassigned unique term identification numbers, when a term alreadyassigned a term identification number is encountered again, either inthe same or in a subsequent document, no new term identification numberis assigned to it.) Rather than storing the term identification numbersthemselves, the differences from the previous term identificationnumbers are stored. For example, the following indicates that Term 1appears 5 times, Term 10 appears 1 time, and so forth:

(1,5) (10,1) (30,2) (50,3) (100,4).

In the preferred embodiment, where the differences or offsets from theprevious term identification numbers are stored, what is actually storedis:

(1,5) (9,1) (20,2) (20,3) (50,4).

By storing the differences instead of the term identification numbers,the numbers to be stored will be considerably smaller. This allows thesurrogate to be compressed by using a variable length encoding of theinteger values. The differences are encoded using Golomb coding.(Golomb, S. W. 1966. Run-length encodings. IEEE Transactions onInformation Theory, vol. 12 no. 3 pp 339-401)

The term counts are encoded in unary, i.e. the number 1 is encoded as 0,2 is encoded as 10, 3 as 110 etc. Someone of ordinary skill in the artwill recognize that other variable length encodings may also be used toencode these values.

By compressing the differences and counts, the document surrogates canbe stored in only 10% of the space required by the original text.Similarly, if one were to store the within document position in thesurrogate, the difference from the previous position would be storedrather than the absolute position. (Thus, a term occurring in positions1, 3, 5, and 10 in a document will have this information stored as 1, 2,2, 5.) As before, the smaller average sizes allow the information to beencoded in fewer bits, thereby saving space.

Thus, in FIG. 8, a surrogate 810 lists a term identification number,“1”, of a first term, Term 1, used in a document 805, and the number ofoccurrences (two) of Term 1 in the document 805. The surrogate 810 thenlists the difference between the term identification number, “1” of thefirst term, and the term identification number “2” of a second term,Term 2, which occurs in the document 805, namely “1”, and the number ofoccurrences (two) for Term 2 in the document 805, reflecting that thatterm is present in the document 805. The surrogate 810 then lists thedifference between the term identification number, “2” of the secondterm, and the term identification number “3” of a third term, Term 3,which occurs in the document 805, namely “1”, and the number ofoccurrences (one) for Term 3 in the document 805, reflecting that thatterm is present in the document 805. Note that the surrogate 810 onlycontains a single entry for Terms 1 and 2, even though the terms occurmore than once in the underlying document 805. Similarly, a surrogate820 for a second document 815 lists the term identification number, “1”,of Term 1, and the number of occurrences (one) of Term 1 in the document815, because Term 1 is present in the Document 815, but the surrogate820 does not list Term 2, because Term 2 is not present. The surrogate820 then lists the difference between the term identification number,“3”, of Term 3, and the term identification number of Term 1, “1”,namely “2”, and the number of occurrences of Term 3, because Term 3 ispresent, and so on.

Terms may be stored in a surrogate in any suitable order, such as butnot limited to alphabetical order. In the preferred embodiment describedhere, the terms are stored in order of term identification number. Inthe preferred embodiment, in order to conserve space, furtherinformation about terms is stored in a lookup table 102 of the typeillustrated in the lower portion of FIG. 7. The location in the lookuptable of information concerning the term of interest may be determinedfrom the term identification number, in that the term lookup table is afixed array and terms are stored in the table in order of the termidentification number. For each term, the term lookup table identifiesthe actual term and contains further information about the term, such asthe location of an inverted term list for the term, the number ofdocuments in the collection in which the term occurs, and the maximumterm frequency scores for the term in any one document in the term's“top” inverted term list, and in any one document in the term's“remainder” inverted term list.

In the system described herein, compressed document surrogates may beutilized to reduce the time required to determine the score for adocument with respect to a given search query. Conventionally, the scorefor a document, with respect to a given search query, is determined bysearching the inverted term lists for all of the terms in the query.Because it is not known prior to beginning such a search which of theterms in the query is in the document, it is necessary to search theinverted term lists for all of the terms in the query to determine thescore for a document. Finding whether a given document occurs in aninverted term list may be a relatively time-consuming process, if thereare many terms in the query.

Inverted term lists, however, may permit a document score to bedetermined more quickly by the use of the document's compressed documentsurrogate. Referring to FIG. 9, a process 500 begins at a step 525 byexamining a compressed document surrogate for a document to be scoredwith respect to a particular search query. A term in the search querywhich occurs in the document is identified by using the compresseddocument surrogate. Then, a step 530 calculates the score resulting fromthe occurrence of the term in the document by consulting, if necessary,a lookup table and/or inverted term list for the term. Then, a step 540determines whether any other terms in the search query, which are foundin the compressed document surrogate, have not yet been analyzed. If allterms in the search query that are found in the compressed documentsurrogate have been analyzed, the process 500 is completed. Otherwise,the process 500 continues by returning to the step 525 to choose thenext term in the search query which occurs in the document and has notyet been analyzed, and then doing the appropriate calculation andadjustment of score.

In the preferred embodiment, at the step 530 it is not necessary toconsult the inverted term list for the term, since the number ofoccurrences of the term in the document is known from the compresseddocument surrogate, and the remaining information necessary to calculatethe document's score may be determined from the term lookup table by useof the term identification number in the compressed document surrogate,without the need to refer to the inverted term list itself.

A further aspect of the system described herein which takes advantage ofcompressed document surrogates to facilitate carrying out search queriesto return documents related to the portion of the temporal document ofinterest to a user may now be described.

The formula used for identifying documents which relate to the portionof the temporal document in which the user has expressed an interest is:${S_{D} = {\sum\limits_{{terms}\quad T}{{W(t)}*{TF}_{TD}*{IDF}_{T}}}},$

The terms in the formula are as defined above.

This formula among others takes advantage of the fact that a “rare” termis a more powerful predictor of document utility than a common term, bygiving greater weight in ranking documents to those that occurrelatively less often in the collection. For example, if a user hasindicated interest in a portion of a temporal document which includesthe phrase “osteoporosis in women”, the term “osteoporosis” alone, if itoccurs in the document collection in fewer documents than the term“women,” may be of more utility as a filter than the term “women.”However, it may also be true that, among documents which refer toosteoporosis, those that also mention women are more likely to be usefulthan those that do not. Hence, the formula does not exclude the commonterm from the search process entirely.

It is possible to reduce the time taken to apply the search querygenerated to identify N documents related to the portion of the temporaldocument in which the user has expressed an interest, by usingcompressed document surrogates.

Referring to FIG. 10, shown is a flowchart of an embodiment of a methodfor using compressed document surrogates to apply a search query toidentify documents related to the portion of the temporal document. Aprocess 600 begins with a step 605 wherein it is determined to beginusing top inverted term lists for the terms in the query.

According to FIG. 10, the process 600 iterates until a sufficient numberof candidate documents for inclusion in the final ranking of N documentsis generated.

The iterative portion of the process 600 begins at a step 610 bychoosing, from among those terms which are in the query, the mostsignificant term whose top inverted term list has not yet been analyzed.Terms may be ranked in order of significance using any one of a numberof measures which will be known to those of ordinary skill in the art.In the preferred embodiment discussed here, the ranking is done by usingthe quantity W(t)*IDF_(T), where W(t) is the weighting function for theterm T which occurs at time t, and IDF_(T) is the inverted documentfrequency for term T:

IDF _(T) =log((N+K ₃)/N _(T))/log(N+K ₄)

where:

N is the number of documents in the collection,

N_(T) is the document frequency of the term T in the collection, whichis the number of documents containing the term T in the collection,

K₃ and K₄ are constants. (K₃ typically may be assigned a value of 0.5,and K₄ 1.0, but these values may be varied without departing from thespirit and scope of the invention.)

This particular formula is by no means the only formula that may be usedto select the order in which terms are analyzed. Other formulae will beapparent to one of ordinary skill in the art.

At a step 615, a top inverted term list for that most significantnot-yet-analyzed term is examined. In the embodiment illustrated herein,the top list contains one thousand documents, but the number ofdocuments may vary according to a variety of functional factors familiarto one of ordinary skill in the art, such as the total number ofdocuments of potential interest.

The process 600 then continues at a step 625 by calculating, for eachdocument D on the top inverted term list for the term T, the scoreS_(TD) resulting from its containing the term, where:

S_(TD)=W(t)*TF_(TD)*IDF_(T), where W(t), IDF_(T) and TF_(TD),Robertson's term frequency for Term T, are as set forth above.

If a document D for which a score S_(D,T) has been calculated has notpreviously been found on an inverted term list in the process 600, thedocument is added to a list L of candidate documents. If the documenthas been found on an inverted term list previously in the process 600,the document's prior score is adjusted by adding S_(D,T) to the priorscore.

After this is done, the process 600 continues at a step 630 bycalculating the maximum number of points that could be scored by adocument not yet found to contain any analyzed term. (That is, adocument that contains all of the desired terms not yet analyzed.) Thatmaximum potential score S_(Max) is the sum, over all the desired termswhose hit lists have not yet been analyzed:S_(Max) = ∑W(t) * TF_(Max) * IDF_(T),

where: TF_(Max)=Robertson's maximum term frequency for Term T

=MAX(N _(TD)/(N _(TD) +K ₁ +K ₂*(L _(D) /L ₀))),

where: N_(TD), L_(D), L₀, and K₁ and K₂ have the values set forth above,and W(t) and IDF_(T) have the value set forth above.

At a next step 635, it is determined whether there are already Ndocuments on the list L whose scores exceed S_(Max), the maximum numberof points that could be accrued by a document not found on any of thetop inverted term lists yet analyzed. If there are N or more suchdocuments, it is unnecessary to look for any further documents bysearching the top inverted term lists of the (relatively lesssignificant) terms not yet analyzed, and a next step 640 in the process600 calculates a final score for all of the already-located documents onthe list L, so that their rankings may be adjusted to account for thedocuments containing the less significant terms, and a final list of thetop N documents may be prepared.

At the step 640, in calculating the final scores for the candidatedocuments on the list L the process 600 may take advantage of thataspect of the system previously discussed which permits the score of adocument to be determined by use of its compressed document surrogate.The process then concludes at a step 645 by ranking the documents on thelist L according to the scores of the documents, and returning as itsresult the N documents which have the highest scores, ranked in order ofthe scores.

If it is determined at the step 635 that there are not N documentsalready found whose scores exceed the scores that could be achieved bynot-yet-located documents, then the process continues at a step 650 todetermine if there are any terms in the search query whose top invertedterm lists have not yet been analyzed.

If the process 600 determines at the step 650 that not all terms havehad their top inverted term lists analyzed, then the process 600continues by returning to the step 611 to begin analyzing the mostsignificant term not yet analyzed.

If all terms in the search query have had their top inverted term listsanalyzed, then the process 600 proceeds to a step 655. When the process600 reaches the step 655 after processing top inverted term lists, it isconcluded that remainder inverted term lists have not yet been analyzed,and the process 600 proceeds to a step 660. (The path the process 600will follow when the step 655 is reached after the remainder invertedterm lists have been analyzed will be discussed below.)

In the process 600 at the step 660 it is concluded that remainderinverted term lists will now be processed, and control passes to thestep 610.

At the step 610, the iterative process of considering the mostsignificant term whose inverted term list has not yet been analyzedbegins again, this time considering the remainder inverted term lists.The process 600 cycles through the remainder inverted term lists atsteps 615, 625 adding documents to the list L, and increasing the scoresof the documents already on the list L, as documents are found on theremainder inverted term lists. As before, after each inverted term listis processed at the step 630 a new S_(Max) is determined. In doing thisfor the remainder term lists, the maximum term frequency scores againmay be determined in the preferred embodiment from the lookup table, butthey are not the same maximum term frequency scores as were used for thetop inverted term lists. Instead, the lookup table maintains a list ofmaximum term frequency scores for terms, for documents found in theremainder lists for the terms.

At the step 635 it is determined whether further inverted term listsneed to be processed, or whether a sufficient number of documents havebeen found with sufficiently high scores that no further lists need besearched.

If it is concluded that a sufficient number of documents withsufficiently high scores as described above have been located, then fromthe step 635 control passes to the step 640, and as described abovefinal scores are calculated, and a final list of N documents with thehighest scores is returned, ranked in order of score.

However, if the process 600 proceeds to complete the iterations throughall of the remainder inverted term lists without generating a sufficientnumber of documents with sufficiently high scores, then after the step635 control passes through the step 650, where it is determined thatthere are no terms left whose remainder inverted term lists have not yetbeen processed, to the step 655, where it is determined that because theremainder term lists have been processed, control is to pass to the step640 to begin the final processing. If the step 640 is reached after theremainder inverted term lists have all been processed, the final scoresof the documents on the list L are calculated, and control passes to thestep 645 to rank the documents that have been located in order, exceptthat the process returns fewer than N documents.

A further aspect relates to resolving the potential capacity problemwhich may occur when multimedia material such as video is communicatedin a digital fashion.

Conventional synchronous multimedia documents (i.e., temporal documentswhich contain two media types such as video and text) contain all thesynchronization information hard-coded in the document. For example, thetext that would scroll in conjunction with a certain video frame or setof frames is predetermined and hard-coded into the multimedia document.When the document is transmitted for viewing, the server ensures thatthe text data is transmitted at the appropriate time with the relatedvideo frames, and the network carries both components of thedocument—video and text—to the user.

This conventional approach to encoding and providing synchronizationinformation requires that the server send all this material to the user.This increases the load on the server and on the network, thus reducingthe number of users who may be serviced at a given time. While this isappropriate if the user is taking advantage of the synchronizedinformation, such as the text which would accompany the video, it isunnecessary if the client uses the information in the synchronizeddocument only sparingly or not at all.

One aspect of the system described herein reduces the load on the videoserver and network by not creating and transmitting the synchronizeddocument to the user from the video server on which the video is storedunless the user requires it. Instead, only the video material is sent tothe user.

In this aspect, it is recognized that, although a search query may berun at any time when a temporal multimedia document such as a video isbeing transmitted and viewed, and although that search query willutilize the close caption text associated with the video, it is notnecessary to create a synchronized document containing all of the closecaption text. Rather, a table may be created containing the text that isin the closed caption, and the associated times at which the text occursin the video, that table may be stored, and that table may be utilizedto create the query when appropriate.

Another aspect of the system described herein permits the use of thesystem with “live” material which is supplied to a user immediately asit is occurring, or with material which the user obtains elsewhere onthe Internet which has not been previously prepared by the system andplaced in a video library to be made available through a video servermaintained in connection with the system. In this aspect, no pre-storedtable can be used to provide the text which corresponds to the portionof the temporal document in which the user has indicated an interest,because the material is being supplied to the user as it is created orobtained from elsewhere on the Internet.

The user may be permitted to select the “live” material in any one of anumber of ways which will be known to one of ordinary skill in the art.In one embodiment, the user may be given a list of “live” documentswhich are available, and permitted to choose one, by clicking on it orindicating his interest in any one of a number of alternative ways whichwill be known to one of ordinary skill in the art. Alternatively, theuser may be invited to search by using search engine or search querytechniques such as will be familiar to one of ordinary skill in the art.Still other methods to permit the user to choose a document will beknown to one of ordinary skill in the art. The user then may view (orlisten to) the temporal document chosen through his work station 2connected to the Internet 5.

In other embodiments, the user may be permitted to obtain material fromelsewhere on the Internet which has not been previously prepared by thesystem and placed in a video library to be made available through avideo server maintained in connection with the system. In one of theseembodiments, the user may be permitted to employ a search engine whichis maintained as part of the system to find and retrieve a document tothe system. The search engine employed may be any one of a number of atype which will be familiar to one of ordinary skill in the art. Theuser then may view (or listen to) the temporal document chosen throughhis work station 2 connected to the Internet 5.

In this aspect, the text associated with the portion of the temporaldocument in which interest has been indicated is obtained by the systemas the document is accessed by the user. For example, in the embodimentwhere the temporal document is video, and close caption information isused as the source of the text, as the video is supplied to the user theclosed caption text is stored in a buffer.

According to one method of implementation, the buffer size may be fixed,at a size sufficient to permit the storage of as many terms as may occurwithin the maximum length of time for which information must be retainedin order to permit a query to be constructed when interest is indicatedby a user. For example, in the embodiment where it is assumed that onlyterms that occur within the 30 seconds prior to the indication ofinterest will be included in the search query, the buffer may be madelarge enough to contain sufficient storage positions to accommodate allterms which may occur in a 30 second interval. In one embodiment, abuffer size of 8 kilobytes is used.

In another embodiment, the buffer size may be varied as necessary sothat there is always sufficient space in the buffer to store all of theterms which have occurred within the maximum length of time for whichinformation must be retained in order to permit a query to beconstructed when interest is indicated by a user. For example, in theembodiment where it is assumed that only terms that occur within the 30seconds prior to the indication of interest will be included in thesearch query, the buffer size may be varied as necessary so that allterms which have occurred within the prior 30 second interval have beenretained.

As time progresses, the terms are stored sequentially in the buffer inthe order in which they occur temporally, with each also having storedthe time at which it occurred. When the last position in the buffer hasbeen filled, the storage then cycles back to the first position in thebuffer, and begins again sequentially, overwriting the terms previouslystored in each position. This process is continued indefinitely, as longas the video lasts. At any time interest is expressed, it will always bepossible to locate all terms required for the query in the buffer, sinceit takes 30 seconds or longer to make one complete storage cycle throughthe buffer. The terms of interest are determined by locating the termswhose associated time values are between the time the signal of interestoccurred, and a time 30 seconds before that. The producer-consumermethod as described in Jeffay, K., “The real-time producer/consumerparadigm: a paradigm for the construction of efficient, predictablereal-time systems,” Proceedings, 1993 ACM/SIGAPP Symposium on AppliedComputing: States of the Art and Practice, pp. 796-804, may be used toprevent the storage of new information in a portion of the buffer whosecontent may be required for the generation of a query.

In another embodiment, the temporal document may be obtained fromanother source on the Web. In this embodiment, the user may be permittedto employ a search engine on his work station 2 connected to theInternet 5 to retrieve and view (or listen to) the temporal documentchosen. The search engine employed may be any one of a number of a typewhich will be familiar to one of ordinary skill in the art. The userthen may view (or listen to) the temporal document chosen through hiswork station 2 connected to the Internet 5. In this embodiment, aplug-in program on the user's workstation 2 may determine the locationon the Internet 5 from which the material has been obtained, and maytransmit that information through the Internet 5 to the QSE server sothat the system may access the material. In this embodiment, the time tat which the indication of interest is given is transmitted from theplug-in program to the QSE server and the QSE server then may determinethe weighting function W(t) and extract the relevant text for the searchquery, so that the material of interest to the user may be determined bythe IR server.

In another embodiment, the plug-in program may not transmit the locationon the Internet 5 from which the material has been obtained, but insteadmay determine the portion of the text which is to form the search queryand the weighting function W(t) itself using the system and may transmitthe weighted search query to the IR server so that the IR server maydetermine the material of interest to the user.

The techniques described herein have been described as applied to atemporal document that is supplied to a user from a server. It will beapparent to one of ordinary skill in the art, however, that the samemethod of analysis of text and use of information retrieval (IR)techniques to identify related material that is applied to such dynamicmaterial can also be applied in other contexts. For example, if a user'sown movement over time within and between programs and other material istreated as if it were itself a temporally-sequenced “program,”context-sensitive help could be provided to a user who sought help, byanalysis of the text which the user had visited over a priorpredetermined sequence of time.

While the invention has been disclosed in connection with the preferredembodiments shown and described in detail, various modifications andimprovements thereon will become readily apparent to those skilled inthe art. Accordingly, the spirit and scope of the present invention isto be limited only by the following claims.

What is claimed is:
 1. A method for finding documents which relate to aportion of a temporal document, comprising: (a) in response to a signalof interest at a particular time during the temporal document,identifying a portion of the temporal document for which relateddocuments are to be found; (b) selecting text associated with theportion of the temporal document identified, by reference to a databasewherein is stored each term in text associated with the temporaldocument, together with a time in the temporal document at which theterm occurs; (c) weighting each term in the text selected by a functionW(t) according to the time t at which the text occurs relative to thetime at which the signal of interest occurs; (d) finding the relateddocuments by use of information retrieval techniques as applied to thetext selected; wherein ${{W(t)} = {Wt}_{1}},\begin{matrix}{{t_{2}(t)} = \quad {\int_{k = 0}^{t}{\left( {1 - {\exp \quad \left( {- t_{1}} \right)}} \right)*\exp \quad \left( {{- t_{1}}k} \right)*}}} \\{\quad {{\left( {1 - {\exp \quad \left( {- t_{2}} \right)}} \right)*\exp \quad \left( {- {t_{2}\left( {t - k} \right)}} \right)},}}\end{matrix}$

where t₁ and t₂ are constants.
 2. The method of claim 1, whereint₁=0.0001 and t₂=0.00025.
 3. The method of claim 2, whereinW(t)=W_(t1,t2) (t) for times from 30 seconds before the signal ofinterest is given until the signal is given, and W(t)=0 for all othertimes.
 4. The method of claim 1, wherein the related documents areaccessed through the Internet.
 5. The method of claim 4, furtherincluding selecting the related documents from among a collection ofdocuments which may be accessed through the Internet, by utilizingdatabases comprising information about the collection.
 6. The method ofclaim 5, wherein the related documents are selected from the collectionaccording to the scores achieved when evaluating documents in thecollection according to a formula giving scores to documents dependingupon the occurrence in the documents of terms which occur in textassociated with the portion of the temporal document identified, whereeach term is weighted by a function W(t) according to the time t atwhich the term occurs relative to the time at which the signal ofinterest occurs.
 7. The method of claim 6, wherein the terms which occurin the text which is associated with the portion of the temporaldocument identified, and the times at which the terms occur, areobtained by reference to the database which contains the terms and thetimes in the video or audio material at which the terms occur.
 8. Themethod of claim 7, wherein the times at which the terms occur aredetermined and stored in the database using the synchronizationinformation contained in the temporal document.
 9. The method of claim8, wherein the temporal document uses the SMIL standard forsynchronization of multimedia elements in a document.
 10. The method ofclaim 7, where a predetermined number of documents, 1000, is selected.11. A method for finding documents which relate to a portion of atemporal document, comprising: (a) in response to a signal of interestat a particular time during the temporal document, identifying a portionof the temporal document for which related documents are to be found;(b) selecting text associated with the portion of the temporal documentidentified, by reference to a database wherein is stored each term intext associated with the temporal document, together with a time in thetemporal document at which the term occurs; (c) weighting each term inthe text selected by a function W(t) according to the time t at whichthe text occurs relative to the time at which the signal of interestoccurs; (d) finding the related documents by use of informationretrieval techniques as applied to the text selected; (e) selecting therelated documents from among a collection of documents which may beaccessed through the Internet, by utilizing databases comprisinginformation about the collection; wherein the related documents areselected from the collection according to the scores achieved whenevaluating documents in the collection according to a formula givingscores to documents depending upon the occurrence in the documents ofterms which occur in text associated with the portion of the temporaldocument identified, where each term is weighted by a function W(t)according to the time t at which the term occurs relative to the time atwhich the signal of interest occurs; wherein the terms which occur inthe text which is associated with the portion of the temporal documentidentified, and the times at which the terms occur, are obtained byreference to the database which contains the terms and the times in thevideo or audio material at which the terms occur; and wherein a scoreS_(D) of a document D in the collection may be determined by creditingthe document D, for each term T in the temporal portion of the documentidentified which occurs in the document D, with an amount proportionalto W(t), to Robertson's term frequency TF_(TD) and to ID_(FT) where W(t)is the weight assigned to a term which occurs at time t relative to thesignal of interest, TF _(TD) =N _(TD)/(N _(TD) +K ₁ +K ₂*(L _(D) /L ₀)),and N_(TD) is the number of times the term T occurs in document D, L_(D)is the length of document D, L₀ is the average length of a document inthe collection of documents indexed, K₁ and K₂ are constants, and ID_(FT) =log((N+K ₃)/N _(T))/log(N+K ₄), and N is the number of documentsin the collection, N_(T) is the number of documents containing the termT in the collection, and K₃ and K₄ are constant.
 12. The method of claim11, wherein K₁ is 0.5, K₂ is 1.5, K₃ is 0.5, and K₄ is 1.0.
 13. Themethod of claim 7, wherein terms which occur in portions of the temporaldocument other than the portion identified are utilized in calculatingthe scores achieved when evaluating documents in the collection.
 14. Themethod of claim 7, wherein the determination of the documents in thecollection which receive the highest scores is carried out usingcompressed document surrogates.
 15. The method of claim 7, wherein thedetermination of the documents in the collection which receive thehighest scores is carried out by a server which is distinct from theserver which receives the signal of interest.
 16. A device for findingdocuments which relate to a portion of a temporal document, comprising:(a) means for identifying a portion of the temporal document for whichrelated documents are to be found, in response to a signal of interestat a particular time during the temporal document; (b) means forselecting text associated with the portion of the temporal documentidentified, by reference to a database wherein is stored each term intext associated with the temporal document, together with a time in thetemporal document at which the term occurs; (c) means for weighting eachterm in the text selected by a function W(t) according to the time t atwhich the text occurs relative to the time at which the signal ofinterest occurs; (d) means for finding the related documents by use ofinformation retrieval techniques as applied to the text selected;wherein ${{W(t)} = {Wt}_{1}},\begin{matrix}{{t_{2}(t)} = \quad {\int_{k = 0}^{t}{\left( {1 - {\exp \quad \left( {- t_{1}} \right)}} \right)*\exp \quad \left( {{- t_{1}}k} \right)*}}} \\{\quad {{\left( {1 - {\exp \quad \left( {- t_{2}} \right)}} \right)*\exp \quad \left( {- {t_{2}\left( {t - k} \right)}} \right)},}}\end{matrix}$

where t₁ and t₂ are constants.
 17. The device of claim 16, whereint₁=0.0001 and t₂=0.00025.
 18. The device of claim 17, whereinW(t)=W_(t1,t2) (t) for times from 30 seconds before the signal ofinterest is given until the signal is given, and W(t)=0 for all othertimes.
 19. The device of claim 16, wherein the related documents areaccessed through the Internet.
 20. The device of claim 19, furtherincluding means for selecting the related documents from among acollection of documents which may be accessed through the Internet, byutilizing databases comprising information about the collection.
 21. Thedevice of claim 20, wherein the related documents are selected from thecollection according to the scores achieved when evaluating documents inthe collection according to a formula giving scores to documentsdepending upon the occurrence in the documents of terms which occur intext associated with the portion of the temporal document identified,where each term is weighted by a function W(t) according to the time tat which the term occurs relative to the time at which the signal ofinterest occurs.
 22. The device of claim 21, wherein the terms whichoccur in the text which is associated with the portion of the temporaldocument identified, and the times at which the terms occur, areobtained by reference to the database which contains the terms and thetimes in the video or audio material at which the terms occur.
 23. Thedevice of claim 22, wherein the times at which the terms occur aredetermined and stored in the database using the synchronizationinformation contained in the temporal document.
 24. The device of claim23, wherein the temporal document uses the SMIL standard forsynchronization of multimedia elements in a document.
 25. The device ofclaim 22, where a predetermined number of documents, 1000, is selected.26. A device for finding documents which relate to a portion of atemporal document, comprising: (a) means for identifying a portion ofthe temporal document for which related documents are to be found, inresponse to a signal of interest at a particular time during thetemporal document; (b) means for selecting text associated with theportion of the temporal document identified, by reference to a databasewherein is stored each term in text associated with the temporaldocument, together with a time in the temporal document at which theterm occurs; (c) means for weighting each term in the text selected by afunction W(t) according to the time t at which the text occurs relativeto the time at which the signal of interest occurs; (d) means forfinding the related documents by use of information retrieval techniquesas applied to the text selected; and (e) means for selecting the relateddocuments from among a collection of documents which may be accessedthrough the Internet, by utilizing databases comprising informationabout the collection; wherein the related documents are selected fromthe collection according to the scores achieved when evaluatingdocuments in the collection according to a formula giving scores todocuments depending upon the occurrence in the documents of terms whichoccur in text associated with the portion of the temporal documentidentified, where each term is weighted by a function W(t) according tothe time t at which the term occurs relative to the time at which thesignal of interest occurs; wherein the terms which occur in the textwhich is associated with the portion of the temporal documentidentified, and the times at which the terms occur, are obtained byreference to the database which contains the terms and the times in thevideo or audio material at which the terms occur; and wherein a scoreS_(D) of a document D in the collection may be determined by creditingthe document D, for each term T in the temporal portion of the documentidentified which occurs in the document D, with an amount proportionalto W(t), to Robertson's term frequency TF_(TD) and to ID_(FT) where W(t)is the weight assigned to a term which occurs at time t relative to thesignal of interest, TF _(TD) =N _(TD)/(N _(TD) +K ₁ +K ₂*(L _(D) /L ₀)),and N_(TD) is the number of times the term T occurs in document D, L_(D)is the length of document D, L₀ is the average length of a document inthe collection of documents indexed, K₁ and K₂ are constants, and ID_(FT) =log((N+K ₃)/N _(T))/log(N+K ₄), and N is the number of documentsin the collection, N_(T) is the number of documents containing the termT in the collection, and K₃ and K₄ are constants.
 27. The device ofclaim 26, wherein K₁ is 0.5, K₂ is 1.5, K₃ is 0.5, and K₄ is 1.0. 28.The device of claim 22, wherein terms which occur in portions of thetemporal document other than the portion identified are utilized incalculating the scores achieved when evaluating documents in thecollection.
 29. The device of claim 26, wherein the determination of thedocuments in the collection which receive the highest scores is carriedout using compressed document surrogates.
 30. The device of claim 22,wherein the determination of the documents in the collection whichreceive the highest scores is carried out by a server which is distinctfrom the server which receives the signal of interest.